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1.  BACKGROUND  AND  RATIONALE 

a.  Results  and  Consensus  of  Conference 


In  late  1972,  we  proposed  a  planning  study  to  determine  the  relative 
feasibility  of  various  approaches  for  developing  a  measurement  system 
for  use  in  determining,  at  any  given  point  in  time,  the  status  of  the 
Navy's  personnel  subsystem.  For  want  of  any  other  existing  label  or 
designation,  we  referred  to  the  proposed  measurement  system  as  a  Naval 
Personnel  Status  Index  (NPSl).  it  was  believed  that  availability  of 
an  NPSl  would  aid  in  coping  with  rapid  changes  confronting  the  Navy, 
changes  which  can  be  expected  to  accelerate  with  the  advent  of  all 
volunteer  forces.  If  developed  in  such  a  way  that  variations  in  its 
magnitude  could  yield  information  quickly  about  the  effects  of  changes 
in  Navy  personnel  policies  and  practices,  an  NPSl  could  provide  a  con¬ 
tinuing  audit  or  tracking  of  the  overall  personnel  condition  of  the 
Navy. 

A  first  step  designed  to  define  the  problem  more  explicitly  took  the 
form  of  an  informal  meeting  among  a  small  number  of  experts  represent¬ 
ing  various  behavioral  science  disciplines  and  the  Navy.  The  following 
persons  participated  in  this  brief  two-day  meeting: 


Nicholas  A.  Bond,  Jr. 
Department  of  Psychology 
Sacramento  State  College 

Glenn  L.  Bryan 
Di rector 

Psychological  Sciences  Division 
Office  of  Naval  Research 

John  P.  Campbell 
Department  of  Psychology 
University  of  Minnesota 

Marvin  Denicoff 
Di rector 

Information  Systems  Program 
Office  of  Naval  Research 

Marvin  D.  Dunnette 
President 

Personnel  Decisions,  Inc. 

Eric  Flamholtz 

Graduate  School  of  Business 

Columbia  University 

Warren  Hirsch 
Department  of  Mathematical 
Statistics 
New  York  University 


Andrea  Mol  berg 
Research  Assistant 
Personnel  Decisions,  Inc. 

Stephan  J.  Motowidlo 
Research  Assistant 
Personnel  Decisions,  Inc. 

Lee  Scchrest 

Depart'.ient  of  Psychology 
Northwestern  University 

Wa  1 1  >ce  S i na i ko 
Research  Study  Director 
Smithsonian  Institution 

Frank  Smith 
Director 

Employee  Attitude  Surveys 
Sears  Roebuck  S  Company 

Gerald  Thompson 
Graduate  School  of  Industrial 
Administration 
Carnegie  Mellon  University 
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A  synopsis  of  this  meeting  and  a  selected  literature  review  of  some 
possible  approaches  to  the  problem  of  developing  the  NPSI  are  reported 
by  Dunnette,  Milkovich,  and  Motowidlo  (1973).  That  meeting  touched  on 
several  issues,  including  such  matters  as:  (a)  whether  or  not  the  in¬ 
dex  should  be  reported  solely  as  a  single  composite  or  reported  as  a 
profile  made  up  of  several  relatively  independent  and  diagnostically 
meaningful  measures;  (b)  how  the  Index  should  be  "validated"  in  order 
to  be  interpretable  according  to  evaluative  connotations;  (c)  how 
"current"  it  would  need  to  be  to  be  maximally  useful;  (d)  whether  it 
should  be  used  operationally  or  for  research  purposes  only;  (e)  what 
units  of  analysis  would  be  most  desirable  and  feasible  (e.g.,  tasks  to 
be  done,  work  groups  or  units,  entire  ships,  etc.);  and,  finally,  (f) 
what  sampling  basis  might  be  utilized  in  order  to  evaluate  properly  the 
personnel  status  of  the  Navy  or  of  any  unit  of  the  Navy  at  any  given 
point  in  time.  After  much  discussion  and  some  exchange  of  post-meeting 
correspondence,  these  issues  were  partially  "settled,"  and  the  consensus 
was  summarized  as  follows  by  Dunnette  et  al  (1973): 

"A  Naval  Personnel  Status  Index  is  seen  as  a  potentially  very  use¬ 
ful  descriptive  measure  to  help  in  tracking  and  evaluating  the 
impact  of  changes  in  Naval  personnel  policies  and  practices  as 
they  are  instituted  over  the  months  and  years  ahead.  As  a  coarse 
indicator  of  change,  the  index  should  not  be  solely  descriptive; 
that  is,  certain  levels  on  the  index  should  clearly  be  indicative 
of  "better"  levels  of  the  personnel  condition  in  the  Navy  and  in 
particular  units  of  the  Navy  than  certain  other  levels  on  the  in¬ 
dex.  Moreover,  as  a  coarse  indicator,  the  index  should  be  capable 
of  being  expressed  as  a  single  number,  but  this  need  not  preclude 
its  being  interpreted  in  profile  form,  according  to  its  several 
components.  In  fact,  its  diagnostic  usefulness  and  interpreta¬ 
tions  related  to  possible  causes  of  fluctuations  in  its  value  will 
most  assuredly  be  enhanced  by  retaining  a  capability  for  profile 
interpretation.  Development  of  the  basic  data  system  necessary 
to  comprise  the  index  should  focus  on  tasks  or  functions  as  pri¬ 
mary  units  of  analysis,  initially,  many  tasks  should  be  consid¬ 
ered,  but  methods  should  be  employed  to  reduce  their  number  to 
manageable  proportion  by  evaluating  their  relative  difficulty, 
criticality,  probable  frequency  of  occurrence;  and,  in  particular, 
the  relative  ease  of  personnel  I ntersubst i tutabi 1  i  ty  within  them. 
Once  such  judgments  have  been  obtained  and  confirmed,  purposive 
sampling  of  units  and  functions  may  properly  be  employed  to  de¬ 
velop  data  gathering  and  reporting  systems  necessary  for  computing 
the  index.  Finally,  use  of  the  index  as  an  operational  measure 
to  direct  managerial  decision  making  should  be  avoided  as  a  means 
of  maintaining  not  only  the  Internal  validity  and  evaluative  mean¬ 
ing  of  the  index,  but  also  the  integrity  of  information  bases  from 
which  it  is  computed.  Thus,  such  an  index  should  be  developed 
primarily  for  use  as  a  research  tool  and  not  primarily  for  use  as 
an  operational  indicator  of  Naval  unit  effectiveness.  To  be  used 
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properly  for  personnel  research  and  the  evaluation  of  the  products 
of  personnel  research,  the  Index  must,  of  course,  be  credible;  and 
the  maintenance  of  Its  credibility  constitutes  a  major  reason  for 
resisting  its  use  in  an  operational  context."  (pp.  11-12) 

In  sum,  conferees  concluded  that  the  NPSI  should: 

(a)  be  expressed  as  a  single  number  but  retain  the  identity  of 
its  significant  component  Indicators  for  possible  diagnostic 
purposes; 

(b)  be  interpretable  in  an  evaluative  sense  instead  of  being 
"just"  descriptive; 

(c)  include  components  with  sufficient  ease  of  quick  accessibility 
to  insure  a  high  degree  of  timeliness; 

(d)  be  used  for  research  purposes  only,  primarily  as  a  means  of 
"evaluating"  the  relative  impact  of  changes  in  personnel 
practices  and  policies  or  of  behavioral  research  interventions; 

(e)  focus  on  tasks  or  functions  as  primary  units  of  analysis;  and, 

(f)  be  credible  and  reasonably  free  from  danger  of  loss  of  credi¬ 
bility  through  gun  decking. 

b.  Results  and  Conclusions  from  Literature  Review 

Discussion  of  the  literature  surveyed  by  Dunnette,  Milkovich,  and  Moto- 
widlo  (1973)  was  organized  within  four  areas:  (a)  concepts  of  organiza¬ 
tional  effectiveness;  (b)  efforts  to  measure  operational  readiness;  (c) 
economic  value  or  human  resource  evaluation;  and  (d)  quality  composites 
such  as  "quality  of  life,"  college  quality,  and  the  like.  All  the  metho¬ 
dologies  reviewed  are  similar  in  that  human  judgments  about  such  matters 
as  effectiveness,  readiness,  or  quality  are  in  some  way  quantified  to 
yield  a  numerical  index  or  series  of  Indicators. 

Of  the  methods  reviewed,  we  decided  that  the  various  approaches  designed 
to  develop  quality  composites  [(d)  above]  seemed  most  likely  to  be  prac¬ 
ticable  for  developing  an  NPSI.  A  central  reason  for  this  choice  of  gen¬ 
eral  strategies  is  that  such  approaches  typically  examine  the  judgments  of 
experts  relatively  early  in  the  developmental  stages  rather  than  seeking 
to  impinge  prior  judgments  or  classification  systems  upon  them,  in  other 
words,  knowledgeable  persons  have  a  crucial  early  role  in  determining 
which  components  should  be  examined  as  possible  quality  indicators.  Their 
judgments  are  used  as  a  means  of  pinpointing  which  measures  make  the  most 
sense  or  may  be  most  important  in  contributing  to  an  overall  index  such 
as  "Quality."  Quantification  procedures  may  then  be  utilized  at  later 
stages  in  the  developmental  process  to  determine  the  psychometric 
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properties  of  the  rationally  selected  components  to  provide  clues  for 
deriving  operational  measures  of  them  and  statistical  procedures  for 
combining  them.  In  a  concluding  statement,  Dunnette  et  al  (1973)  ex¬ 
pressed  the  relative  advantages  of  the  "quality"  approach  for  develop¬ 
ing  an  NPSi  as  follows: 

"The  task  of  deciding  what  elements  contribute  to  one's  'quality' 
of  life  seems  .  .  .  [to  be]  a  matter  about  which  some  reasonable 
consensus  ought  to  be  possible.  Why  not,  then,  simply  gather 
together  various  groups  of  Naval  experts  and  work  with  them  to 
name  and  to  define  the  various  indicators  of  good  and  poor  per¬ 
sonnel  status?  ... 

"Ways  of  systematically  gathering  such  indicators  might  then  be 
developed  and  implemented  and  their  relative  levels  and  variabil¬ 
ity  recorded  and  studied  over  time."  (p.  kO) 

c .  Assessment  of  Quality  Indexing  Methodology  Properties  Desired  for 
an  NP$I 

The  major  discrepancy  between  the  approach  being  advocated  above  and  the 
points  of  consensus  derived  by  our  conferees  about  the  desirable  proper¬ 
ties  of  the  proposed  NPSI  involves  the  units  of  analysis  to  be  used.  Our 
literature  search  has  led  us  to  be  pessimistic  about  the  feasibility  of 
being  able  to  synthesize  successfully  organizational  indicators  from  a 
data  base  focusing  on  tasks  or  functions  as  the  primary  units.  We, 
therefore,  laid  aside  (for  the  time  being,  at  least)  our  initial  inten¬ 
tion  to  focus  on  such  molecular  elements  as  tasks  or  functions  and  de¬ 
cided  to  focus  instead  upon  a  more  molar  unit— the  ship--as  the  primary 
unit  for  initial  study;  i.e,,  the  unit  for  which  we  hope  to  derive  a 
first  "experimental"  NPSI. 

With  regard  to  the  other  desirable  NPSI  properties  decided  upon  by  con¬ 
ferees,  the  "quality"  methodology  meets  them  in  the  following  ways: 

(a)  Single  number  vs.  Profile.  By  asking  Naval  experts  to  desig¬ 
nate  those  factors  they  regard  as  Important  Indicators  of  a 
ship's  overall  personnel  status,  we  are  assured  that  their 
primary  focus  wl  1 1  be  on  the  notion  of  a  single  composite  or 
overall  index;  yet,  we  will  learn  also  about  components  that 
are  perceived  as  Important  contributors  to  such  an  overall 
evaluation.  Decisions  about  how  these  components  relate  to 
each  other  and  how  they  may  reasonably  be  combined  into  a 
single  composite  will  be  made  by  utilizing  policy  capturing 
methodology.  [Details  of  policy  capturing  methodology  and 
how  we  propose  to  use  it  in  providing  the  basis  for  these  de¬ 
cisions  are  outlined  in  Part  il  of  this  report.] 
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(b)  Descriptive  vs.  Evaluative.  The  quality  Index  methodology 
obviously  yields  judgments  based  on  an  evaluative  distinc¬ 
tion  between  "good"  personnel  status  and  "poor"  personnel 
status.  Moreover,  the  policy  capturing  methodology  assures 
that  any  components  which  contribute  nothing  or  llt'tle  to 
such  overall  evaluative  judgments  will  be  discovered  and  can 
be  deleted  based  on  results  of  the  policy  capturing  data 
analyses.  As  still  another  way  of  assuring  that  components 
are  selected  Initially  which  do  possess  definite  evaluative 
connotations,  we  also  have  secured  ratings  of  the  relative 
importance  to  overall  personnel  status  of  each  of  the  compo¬ 
nents  suggested.  These  ratings  are  described  in  detail  in 
later  sections  of  this  report. 

(c)  Accessibility  of  Measures.  As  will  be  seen  later  In  this  re¬ 
port,  we  also  have  based  selection  of  initial  components  on 
experts1  ratings  of  the  relative  accessibility  of  measures  of 
the  components  and  upon  ratings  of  their  relative  generality 
across  all  types  of  Naval  vessels.  (For  example,  we  seek 
components  that  are  easily  accessible  on  all  ships,  not  just 
small  ones,  large  ones,  or  those  unique  to  any  other  particular 
subset.) 

(d)  Research  vs.  Operational  Use.  We  have  retained  our  intention 
that  the  NPSI  will  be  used  for  research  purposes  only.  This 
requirement  is  only  relevant  to  the  quality  index  methodology 
in  that  a  wider  range  of  possible  components  are  likely  to  be 
suggested  than  might  be  the  case  if  our  expert  panel  were 
considering  an  NPSI  which  might  ultimately  be  intended  for 
operational  uses. 

(e)  Unit  of  Analysis.  As  already  noted,  we  have  altered  our  Inl- 
tial  plans  by  moving  to  a  molar  level  (J.e.,  "ship")  as  the 
unit  of  analysis  instead  of  focusing  on  the  molecular  level 
of  tasks  or  functions.  [We  do  believe,  of  course,  that  judg¬ 
ments  about  the  components  entering  Into  the  overall  personnel 
status  of  a  ship  definitely  imply  attention  to  the  accompllsh- 
ment  on  that  ship  of  Its  Important  missions,  tasks,  and  func¬ 
tions,  but  these  estimates  enter  indi rectly  instead  of  directly 
Into  the  formulation  of  the  NPSI .] 

(f)  Credibility  and  Robustness.  Credibility  of  the  NPSI  and  the 
components  comprising  it  are,  of  course,  assured  by  the  very 
nature  of  the  quality  indexing  methodology.  As  a  further  means 
of  avoiding  easily  "faked"  or  error  prone  measures,  our  initial 
selection  of  components  Is  also  based,  in  part,  on  experts*  rat 
Ings  of  their  relative  reliability  and  "fudgeabl I i ty ."  These 
scales  and  results  of  ratings  with  them  are  described  in  later 
sections  of  this  report. 
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On  the  following  pages,  we  outline  In  Section  2  the  nature  of  policy 
capturing  methodology  and  our  rationale  for  using  it  in  later  stages  of 
NPSI  development;  in  Section  3,  we  describe  Initial  NPSI  developmental 
steps  taken  during  workshop  meetings  held  with  a  group  of  Naval  officers 
in  Monterey  and  results  of  analyzing  Information  gained  during  those 
meetings;  and,  in  Section  4,  we  describe  our  next  step— to  apply  the 
policy  capturing  methodology  for  the  purpose  of  inferring  the  Importance 
of  each  component  to  NPSI  measurement  from  the  point  of  view  of  expert  - 
enced  Naval  officers,  in  Section  4,  we  also  recommend  a  post-policy 
capturing  strategy  for  the  final  development  and  validation  of  an  NPSI. 
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2.  THE  POLICY  CAPTURING  METHODOLOGY 
a.  General  Description 

One  possibility  for  moving  forward  on  the  choice  of  component 'measures 
for  a  Naval  Personnel  Status  Index  (NPSI)  lies  in  the  recently  developed 
policy  capturing  methodology.  As  we  shall  see,  policy  capturing  and 
related  techniques  can  provide  a  rational  basis  for  weighting  NPSI  com* 
ponents  to  reflect  experienced  Naval  officers1  wisdom.  To  explain  ade¬ 
quately  how  policy  capturing  techniques  will  aid  in  developing  the  NPSI, 
first  we  should  briefly  describe  this  methooology.  Then  we  will  demon¬ 
strate  its  applicability  to  NPSI  development. 

A  number  of  policy  capturing  methods  (e.g.,  Judgment  Analysis--Christal , 
1963,  1968;  Madden,  1983;  Naylor  and  Wherry,  1965)  have  an  important 
objective  in  common— to  develop  models  to  represent  the  way  persons 
weight  information  or  cues  in  making  global  Judgments  about  some  set  of 
stimuli.  For  example,  when  a  supervisor  rates  the  overall  job  perfor¬ 
mance  of  a  number  of  subordinates  based  on  his  knowledge  of  their  work 
in  a  variety  of  areas,  policy  capturing  techniques  might  be  used  to  dis¬ 
cover  the  strategies  he  used  in  combining  his  knowledge  of  their  perfor¬ 
mance  in  different  Job  areas  to  arrive  at  his  overall  impressions  of 
their  performance.  Another  example:  A  member  of  a  selection  board 
charged  with  screening  candidates  for  undergraduate  enrollment  at  a  col¬ 
lege  has  available  to  her  six  test  scores  along  with  ratings  of  potential 
supplied  by  the  candidates1  references.  The  policy  capturing  technique 
can  discover  this  person's  "policy"  of  selecting  undergraduates  by  pin¬ 
pointing  the  cues  she  depends  most  heavily  upon  in  making  decisions  about 
the  candidates. 

In  fact,  policy  capturing  techniques  can  specify  a  rater's  selection 
strategy  or  policy  any  time  that  rater  makes  a  single  global  judgment 
about  each  of  a  series  of  stimuli  based  on  two  or  more  quantifiable 
pieces  of  information  (cues)  for  each  stimulus.  In  general,  it  is  con¬ 
venient  to  generate  a  rater's  policy  by  performing  a  multiple  regression 
analysis  with  cues  as  Independent  variables  and  the  rater's  global  judg¬ 
ments  as  the  dependent  variable.  The  vector  of  regression  weights  rep¬ 
resents  the  rater's  policy.  Each  of  these  weights  may  be  thought  of  as 
an  importance  Index.  That  is,  a  relatively  high  regression  weight  for 
a  given  cue  implies  that  the  cue  is  important  to  the  rater  for  making 
judgments  about  dependent  variable  stimuli.  Smaller  regression  weights 
indicate  cues  of  lesser  importance  to  the  rater.  Thus,  a  rater's  policy 
or  the  way  he  uses  cue  information  to  form  global  impressions  can  be 
captured  in  the  form  of  a  multiple  regression  equation  which  relates  to 
his  use  of  cues  in  making  these  overall  judgments. 

Furthermore,  the  size  of  the  multiple  correlation  between  cues  and  the 
overall  judgments  indicates  the  degree  to  which  the  rater's  evaluations 
are  perfectly  predictable  by  linear  regression.  Two  factors  cause  this 
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multiple  R  to  be  less  than  I .00 — a  conflgural  or  nonlinear  rating  strat¬ 
egy,  or  an  inconsistent  policy.  Goldberg  (1970)  and  Slovlc  and  Lichten¬ 
stein  (1971),  among  others,  have  concluded  that  raters  are  very  seldom 
meaningfully  conflgural  in  their  rating  strategies.  That  Is,  a  linear 
model  has  accounted  for  almost  all  of  the  variance  in  judgments ‘across  a 
variety  of  rating  tasks.  Consequently,  a  vast  majority  of  the  difference 
between  obtained  R  and  1.00  can  be  expected  to  be  related  to  inconsis¬ 
tency  In  rater  policy.  This  result  can  be  used  to  assess  the  consistency 
of  a  rater  group's  composite  policy.  A  relatively  low  R  for  a  group  of 
raters  indicates  that  the  group's  composite  policy  is  inconsistent.  This 
in  turn  implies  that  different  raters  In  the  group  are  using  different 
policies.  Conversely,  a  relatively  high  R  for  a  rater  group  suggests 
that  the  raters  in  the  group  have  similar  policies. 

b.  Rationale  for  Use  of  Policy  Capturing  in  NPSI  Development 

We  submit  that  policy  capturing  provides  a  very  appropriate  way  to  obtain 
a  list  of  component  measures  potentially  useful  for  an  NPSI  composite. 

By  gathering  information  about  what  cues  Naval  officers  weight  heavily  when 
assessing  the  personnel  status  of  a  ship,  we  can  draw  on  the  wisdom  and 
experience  of  persons  who  are  very  familiar  with  the  personnel  subsystems 
of  ships.  In  a  sense,  we  are  formalizing  and  systematizing  that  Naval 
officer  wisdom  when  we  capture  the  policy  of  officers  who  evaluate  the 
personnel  status  of  a  number  of  ships  based  on  a. variety  of  Information 
they  possess  about  each  of  these  ships.  The  way  experienced  Naval  offi¬ 
cers  use  and  combine  this  information  about  ships'  personnel  subsystems 
will  provide  important  clues  about  what  types  of  measures  should  most 
legitimately  be  placed  into  an  NPSI. 

To  obtain  a  clearer  idea  of  advantages  to  be  derived  from  a  policy  cap¬ 
turing  analysis,  let's  contrast  the  policy  capturing  approach  for  aiding 
In  the  identification  of  component  measures  with  the  strategy  employed 
by  General  Electric  in  developing  the  Employee  Relations  Index  (ERI) 
(Merrihue  and  Katzeli,  1955),  an  index  similar  in  meaning  and  In  purpose 
to  the  NPSI.  GE  researchers  spent  relatively  little  time  at  the  con¬ 
ceptual  tasks  of  selecting  ERI  component  measures  and  identifying  areas 
from  which  component  measures  should  be  obtained  or  developed.  Instead, 
they  invested  considerable  time  Insuring  that  the  components  they  selec¬ 
ted  correlated  well  with  each  other  and  that  the  unit  weighted  composite 
ERI  correlated  "properly"  with  organizational  outcome  variables  such  as 
productivi ty. 

Although  the  validation  efforts  just  mentioned  are  legitimate  for  devel¬ 
oping  a  meaningful  and  useful  indicator  of  personnel  status,  we  feel  that 
it  is  a  mistake  to  deemphasize  the  crucially  important  early  component 
generation  phase.  It  is  very  Important  to  exhaust  as  completely  as  possi¬ 
ble  the  personnel  status  domain  by  generating  a  wide  range  of  ideas  for 
components  before  making  assumptions  about  what  components  might  be  more 
important  than  others. 
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Also,  we  suggest  that  the  GE  approach  treated  too  lightly  the  Important 
job  of  conceptually  evaluating  the  legitimacy  and  the  Importance  of  these 
potential  component  measures.  NPSI  development  should  Include  a  thought¬ 
ful  evaluation  of  each  potential  component  measure's  conceptual  suita¬ 
bility  for  inclusion  In  an  NPSI.  Furthermore,  this  conceptual  analysis 
of  NPSI  component  content  should  be  accomplished  before  launching  into 
the  empirical  validation  of  specific  measures  for  an  NPSI.  We  feel  that 
the  policy  capturing  methodology  provides  such  a  means  for  conceptually 
evaluating  components  for  NPSI  measurement.  Attending  to  results  of  the 
policy  capturing  procedure  should  insure  that  final  NPSI  components  will 
be  measuring  content  areas  which  informed  persons  consider  very  important 
for  determining  the  state  of  personnel'  subsystems. 

A  related  reason  for  performing  a  policy  capturing  analysis  in  this  pro¬ 
ject  is  to  screen  the  large  number  of  potentially  important  and  valid 
NPSI  components.  Policy  capturing  results  can  provide  an  unobtrusive 
measure  of  the  importance  of  each  component  (cue  variable)  to  personnel 
status  from  the  perspective  of  experienced  Naval  officers.  Those  compo¬ 
nents  which  are  not  weighted  heavily  in  a  policy  capturing  analysis  can 
be  eliminated  from  further  consideration  as  part  of  the  NPSI.  By  elimi¬ 
nating  through  a  policy  capturing  procedure  components  which  are  not  im¬ 
portant  indicators  of  personnel  status  in  officers'  minds,  we  will  then 
be  able  to  spend  more  time  investigating  measurement  problems  for  each 
component  seen  as  important  for  describing  personnel  status.  And,  we  can 
forget  about  component  measurement  problems  in  areas  seen  as  not  so  Impor¬ 
tant  for  an  NPSI  to  tap. 

A  final  contribution  the  policy  capturing  methodology  can  make  is  to  pro¬ 
vide  an  estimate  of  the  unanimity  of  opinion  among  officers  with  regard 
to  the  way  they  use  information  from  components  or  cues  to  make  overall 
judgments  about  personnel  status.  One  method  with  this  capability,  Judg¬ 
ment  Analysis  (JAN),  groups  together  raters  utilizing  similar  policies. 

The  grouping  process  results  in  information  about  the  number  and  nature 
of  separate  policies  within  a  rater  sample.  That  is,  from  a  group  of 
raters  each  of  whom  rates  the  same  set  of  stimuli  and  uses  the  same,  cues 
as  other  raters,  JAN  can  Identify  distinct  rater  subgroups  relatively 
homogeneous  In  terms  of  the  weights  they  place  on  cues.  Also,  each  sub¬ 
group's  composite  policy  can  be  derived  from  JAN.  In  addition,  Naylor  and 
Schenck  (1966)  have  suggested  using  the  index  pm  to  assess  the  amount  of 
agreement  between  two  raters,  pm  provides  an  index  of  Interrater  agree¬ 
ment  in  terms  of  their  poi icies  rather  than  their  actual  ratings.  Either 
of  these  methods  will  be  appropriate  for  identifying  disagreement  in 
"policy"  among  Naval  officer  raters. 

In  summary,  policy  capturing  provides  a  naturalistic  method  for  assessing 
the  way  experienced  Naval  officers  use  Information  pertinent  to  the  per¬ 
sonnel  subsystem  of  ships  to  make  judgments  about  the  status  of  those 
personnel  subsystems.  By  studying  via  policy  capturing  the  way  Nava) 
officers  make  these  judgments  given  the  information  they  have  available, 
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we  can  assess  the  Importance  of  each  piece  of  information  used  by  these 
officers.  This  analysis  will  in  turn  help  first  to  eliminate  from  fur¬ 
ther  consideration  for  an  NPSI,  those  information  sources  which  are  seen 
uniformly  as  unimportant  Indicators  of  the  state  of  personnel  subsystems 
on  ships.  And,  second,  policy  capturing  results  will  suggest  the  degree 
of  homogeneity  of  officer  opinions  concerning  the  value  different  infor¬ 
mation  sources  have  for  measuring  personnel  status. 

Having  provided  a  general  background  statement  about  policy  capturing  and 
its  applicability  to  NPSI  development,  we  now  describe  more  specifically 
the  approach  we  favor  for  progressing  toward  the  development  of  a  useful 
and  valid  NPSI.  In  the  course  of  this  discussion,  we  report  results  of 
a  pilot  investigation  workshop  held  with  Naval  officers.  The  purpose  of 
the  workshop  was  to  generate  potential  components  for  an  NPSI  composite. 
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3.  INITIAL  NPS I  DEVELOPMENT:  MONTEREY  WORKSHOPS 
a.  Workshop  Procedures 

An  obvious  first  step  toward  selecting  components  for  an  NPSI  'is  to  gen¬ 
erate  a  list  of  potentially  useful  component  measures.  To  accomplish 
this,  two  PDI  staff  professionals  organized  and  led  a  two-day  workshop 
designed  to  generate  such  a  list.  Approximately  23  Naval  officers  attend¬ 
ing  the  Navy  Postgraduate  School  in  Monterey,  California  participated  In 
the  workshop  sessions.  (See  Appendix  1  for  a  copy  of  the  workshop  booklet 
containing  a  participant  roster  and  the  schedule  of  workshop  events.)  in 
addition,  a  small  group  of  Instructors1 at  the  Postgraduate  School  attended 
some  or  all  of  the  sessions.  The  breadth  and  amount  of  experience  po¬ 
ssessed  by  this  group  seemed  to  be  adequate  for  our  purposes.  For  example, 
the  workshop  group  contained  officers  with  shipboard  experience  in  small 
patrol  vessels,  destroyers,  cruisers,  and  carriers. 

Furthermore,  although  no  officer  of  captain  rank  or  above  attended  the 
workshops,  we  felt  that  the  group's  experience  level  was  high  in  terms 
of  knowledge  about  a  wide  variety  of  potential  NPSI  component  measures. 
Most  officers  were  quite  familiar  with  such  data  systems  as  NEC  and  3M. 
Also,  most  participants  were  knowledgeable  about  the  configuration  of  a 
variety  of  Bureau  of  Naval  Personnel  data  potentially  valuable  for  an  NPSI 
composite.  Finally,  all  officers  knew  a  considerable  amount  about  data 
routinely  kept  aboard  individual  ships;  and,  they  had  a  good  feel  for 
possible  ways  to  combine  existing  data  to  yield  useful  component  measures. 

During  the  introduction  period,  PDI  staff  members  discussed  NPSI  project 
progress  and  outlined  the  upcoming  workshop  activities.  Participants 
were  then  divided  into  two  subgroups  to  generate  ideas  for  potential  com¬ 
ponent  measures.  Officers  in  one  of  the  subgroups  were  encouraged  to  re¬ 
call  ships  they  had  been  assigned  to  or  were  familiar  with  and  then  to 
think  about  the  status  of  that  ship's  personnel  subsystem.  Then,  these 
officers  were  asked  to  record  the  cues,  Information,  or  factors  which  led 
them  to  assess  the  personnel  status  of  that  ship  the  way  they  did.  This 
method  resulted  in  the  generation  of  20-30  components  potentially  appli¬ 
cable  to  NPSI  measurement.  The  PDI  leader  of  the  other  subgroup  requested 
that  his  officers  generate  potential  NPSI  component  measures  directly. 
Thus,  officers  in  this  subgroup  introduced  component  measure  ideas  directly 
to  others  in  the  subgroup.  Then,  each  suggestion  was  discussed  and  even¬ 
tually  accepted,  rejected,  or  refined  by  other  group  members.  This  pro¬ 
cedure  also  resulted  in  20-30  ideas  for  NPSI  components. 

The  two  PDI  staff  members  then  pooled  the  Ideas  they  had  received  from 
their  groups.  There  was  considerable  overlap  between  the  two  groups'  out¬ 
put  in  terms  of  the  kinds  of  components  suggested  and  even  the  actual 
measures  proposed  for  each  component.  The  pooling  operation  yielded  29 
measures  or  areas ^  seen  as  important  for  NPSI  measurement. 


*See  Appendix  2  for  the  NPSI  component  list.  Where  possible,  the  PDI  staff 
members  specified  the  way  a  component  was  meant  to  be  measured,  it  was  impossi¬ 
ble  to  know  exactly  how  some  of  the  components  might  be  measured,  however.  In 
these  cases,  we  stated  as  precisely  as  possible  the  content  or  nature  of  the 
component . 
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These  29  components  were  named  and  listed  In  preparation  for  the  next 
day's  rating  session.  The  second  day  of  the  workshop  was  devoted  to  two 
rating  tasks.  First,  officer  participants  were  asked  to  evaluate  each 
component  on  five  criteria  of  Importance  and  utility  for  use  in  an  NPSI 
composite.  (See  Appendix  3  for  an  example  of  the  rating  task.)*  The  five 
criteria  were:  Importance;  Reliability;  Accessibility;  General izabl 1 1 ty; 
and  "Fudgeabl 1 1 ty."  The  purpose  of  obtaining  these  criterion  ratings 
was  to  assess  officers'  perceptions  of  the  "goodness"  of  each  component. 
These  opinions  of  component  quality  on  a  number  of  criteria  enabled  us  to 
identify  several  components  which  showed  promise  for  contributing  to  an 
NPSI  composite. 

The  second  rating  task  required  each  officer  to  respond  to  one  of  two 
formats  designed  to  estimate  the  Intercorrelations  among  component  mea¬ 
sures.  That  is,  instead  of  obtaining  the  empirical  relationships  among 
components  by  actually  obtaining  measures  from  a  large  number  of  ships 
and  then  computing  their  Intercorrelations,  we  asked  officers  to  provide 
us  with  estimates  of  these  intercorrelations.  One  of  the  formats  required 
raters  to  picture  a  ship  wh!ch  stood  extremely  favorably  along  a  particu¬ 
lar  component  (1  of  the  29).  The  rater  was  then  asked  to  rate  from  I 
(extremely  unfavorable)  to  9  (extremely  favorable)  the  probable  standing 
of  that  ship  along  the  other  28  components.  The  other  format  was  identi¬ 
cal  except  that  the  rater  was  asked  to  picture  a  ship  which  stood  ex¬ 
tremely  unfavorably  on  a  particular  component.  (See  Appendix  3  for  “he 
two  rating  protocols.) 

This  rating  task  provided  a  relatively  uncomplicated  method  for  obtaining 
each  rater's  estimate  of  the  relationships  among  components.  Furthermore, 
we  received  these  estimates  from  a  number  of  officers  independently  and 
from  two  separate  rating  formats. 

b .  Workshop  Results 

I.  Ratings.  Results  of  the  criterion  ratings  appear  in  Table  1.  It 
is  of  Interest  to  note  that  the  general  importance  level  of  the 
components  was  seen  as  quite  high.  Only  six  components  have  mean 
importance  ratings  of  below  3*00  on  a  scale  of  1  to  5 •  Also,  the 
general izabl I i ty  results  were  encouraging.  All  means  on  this 
criterion  were  near  2.00 — fully  general izable.  Thus,  according  to 
these  Nava]  officers,  there  should  be  few  problems  obtaining  com¬ 
ponent  scores  across  different  ship  types. 

We  used  these  criterion  results  to  select  from  the  total  list  of 
29  those  components  which  seemed  most  promising  for  providing  high 
quality  NPSI  measurement.  These  14  component  measures  are  now  In 
the  process  of  being  studied  in  the  policy  capturing  phase  of  our 
research.  Here  are  the  specific  criteria  used  in  selecting  the 
14  components  for  inclusion  in  the  policy  capturing  rating  task. 
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(a)  importance.  For  the  most  part,  components  rated  as  rela¬ 
tively  important  were  selected  for  the  policy  capturing 
rating  task.  However,  two  components  (Sack  Time  and  Dis¬ 
charge)  were  included  as  "dummy  variables."  That  is,  these 
two  variables  were  seen  as  not  very  Important  by  officers 
in  the  workshop.  If  the  policy  capturing  analyses  suggest 
that  they  are  very  important  In  terms  of  affecting  ratings 
of  personnel  status,  we  wl 1 1  have  reason  to  question  the 
policy  capturing  results.  Thus,  these  two  components  have 
been  Included,  in  a  sense,  to  check  on  the  policy  capturing 
procedure. 

(b)  General izabi 1  i ty.  Components  were  required  to  be  potentially 
general izabie  across  ship  type. 

(c)  Reliability,  Accessibility,  and  "Fudgeabi 1 i ty ."  We  attempted 
to  select  components  which  showed  up  relatively  favorably  In 
these  three  areas  (except  for  the  two  "dummy  variables"). 

(d)  Range  of  content.  Components  were  selected  such  that  the  total 
group  of  l*»  represented  a  wide  range  of  content  (e.g.,  disci¬ 
pline,  crew  satisfaction)  and  diverse  ways  of  measuring  com¬ 
ponents  (e.g.,  objective  measures,  ratings  by  crew  members). 

ii.  Perceived  relationships  among  components.  Table  2  displays  the 
results  of  rating  task  #2  designed  to  estimate  the  correlations2 * * * * 7 
among  components.  We  also  obtained  a  crude  estimate  of  inter¬ 
rater  agreement  with  respect  to  the  independent  rater  estimates 
of  the  relationships  among  components.  Three  intraclass  correla¬ 
tion  coefficients  (R|)  (Haggard,  1958)  were  computed — one  each  for 
ratings  provided  on  the  "favorable"  rating  task,  the  "unfavorable" 
rating  task,  and  the  two  combined. 3  The  results:  R|  (favorable)  - 
,kk\  Rj  (unfavorable)  ■  .53;  R|  (combined)  -  .63.  The  R|S  provide 

2Ther.e  "correlations"  were  obtained  by  f i rut  applying  an  arbitrary  linear 

transformation  to  the  ratings.  For  the  task  in  which  raters  were  to  imagine  a 

ship  with  an  extremely  favorable  standing  along  a  component  and  then  to  estimate 

the  same  ship's  probable  standing  on  other  components,  a  rating  of  9  (extremely 

favorable)  was  transformed  to  +1.00.  The  other  transformations  were  8  ■  +.75; 

7  -  +.50;  6  =  +.25;  5  -  0;  k  -  -.25;  3  -  -.50;  2  -  -.75;  1  -  -1.00.  For  the 
task  in  which  raters  were  to  imagine  a  ship  with  an  extremely  unfavo rable  stand¬ 
ing  along  a  component  and  then  to  estimate  the  same  ship's  probable  standing 
along  the  other  components,  the  transformations  were,  of  course,  reversed.  That 
is,  1  =  +1.00;  2  ■  +.75,  etc.  Then,  all  estimates  of  correlations  among  compo¬ 
nents  were  averaged  to  obtain  the  coefficients  appearing  in  Table  2.  We  recog¬ 
nize  that  this  is,  at  best,  an  extremely  crude  approach  to  approximating  the 
correlations  among  these  variables.  Nonetheless,  it  does  give  a  first  look  at 
how  experienced  Naval  officers  perceive  these  variables  to  covary.  As  such,  the 
matrix  provides  a  better  basis  for  concocting  the  hypothetical  ships  in  the  policy 
capturing  phase  than  would  be  provided  by  our  own  a  priori  judgments. 

3For  the  "combined"  analysis,  ratings  from  the  "unfavorable"  task  were  first 
reflected.  That  is,  Is  were  transformed  to  9s,  2s  to  8s,  etc. 


TABLE  2 

MEAN  ESTIMATES  OF  INTERRELATIONSHIPS  AMONG  29  COMPONENTS* 
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a  rough  idea  of  the  variation  in  the  estimates  of  interrelation** 
ships  among  components  provided  by  different  raters  for  the  same 
component  pairs,  compared  to  the  overall  variation  In  the  estl- 
mates  across  all  raters  and  all  component  pairs.  The  relatively 
high  positive  RjS  suggest  that  officer  raters  agreed  substantially 
among  themselves  about  the  relative  magnitudes  of  correlations 
among  these  components. 

Although  the  arbitrary  linear  transformations  discussed  in  Foot¬ 
note  2  make  untenable  a  serious  consideration  of  absolute  correla¬ 
tion  size,  it  is  of  interest  to  note  that  many  of  the  estimated 
relationships  among  components  are  near  zero  or  only  moderately 
positive.  This  pattern  suggests  (provided  the  officer  raters  are 
right)  that  it  may  be  unrealistic  to  expect  an  NPSI  composite  to 
consist  of  a  number  of  highly  related  components,  all  measuring 
the  same  narrow  unidimensional  construct.  An  NPSI  may  need  to  be 
multidimensional  in  nature  to  sample  adequately  the  multifaceted 
domain  legitimate  for  an  NPSI  to  tap.  TH  complexity  of  a  composite 
NPSI  will  increase  the  problems  In  val Muring  It,  but  the  complex 
multidimensionality  of  such  a  composite  should  also  increase  the 
richness  of  the  information  provided  by  the  index. 

iii.  Factor  analysis.  As  an  extremely  tentative  step  toward  exploring 
the  possible  dimensionality  of  the  measurable  personnel  status 
domain,  we  factor  analyzed  the  correlation  matrix  generated  by 
the  estimates  of  the  relationships  among  components  provided  by 
officers  in  rating  task  HI.  For  this  analysis,  we  used  only  those 
l^t  components  to  be  included  in  the  policy  capturing  phase.  Table 
3  depicts  the  1 4  X  14  correlation  matrix  representing  officers1' 
estimates  of  the  relationships  among  those  14  components.  We  pre¬ 
sent  the  factor  analysis  results  In  Table  4.  To  obtain  these  re¬ 
sults,  a  principal  components  factor  analysis  first  was  performed. 
Then,  the  first  five  factors  were  rotated  using  the  varimax  cri¬ 
terion.  These  rotated  factors  appear  in  Table  4.  Although  we 
must  keep  in  mind  the  source  of  the  original  correlations  on  which 
this  analysis  is  based,  the  results  do  suggest  very  meaningful 
clusters  of  components. 

Factors 


I  -  Fitness  and  Readiness  of  individuals 
II  -Discipline 

III  -  Crew  Member  Attitude  Toward  Officers  and  the  Navy 
IV  -  Free  Time  Activities 
V  -  Manning  Level 

Again,  it  must  be  emphasized  that  these  factor  analysis  results  are 
based  on  data  from  a  correlation  matrix  generated  in  a  very  unusual 
manner.  Yet,  the  results  do  suggest  that  It  may  be  meaningful  and 
useful  to  represent  the  personnel  status  construct  in  terms  of  mul¬ 
tiple  dimensions  of  the  type  appearing  in  the  above  factor  analysis 
results . 


MEAN  ESTIMATES  OF  INTERRELATIONSHIPS 
AMONG  COMPONENTS  SELECTED  FOR  THE  POLICY  CAPTURING  TASK* 


*See  Footnote  2,  page  14,  for  an  explanation  of  how  these  correlations  were  computed. 
**Decimal  points  omitted. 


VAR I MAX  FACTOR  LOADING  MATRIX  FOR  THE  14  COMPONENT  MEASURES 
_ SELECTED  FOR  THE  POLICY  CAPTURING  TASK 
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4.  NEXT  STEPS:  POLICY  CAPTURING  JUDGMENTS  ON  MOCK  SHIPS 

a.  Generating  Mock  Ships 

We  will  ask  approximately  50  Naval  officers  to  participate  in' the  policy 
capturing  rating  project.  The  procedure  will  require  each  officer  to 
study  individual  mock  ships,  each  ship  characterized  by  scores  on  the  14 
components  identified  earlier.  Raters  will  be  instructed  to  evaluate 
each  "ship's"  personnel  subsystem  in  tens  of  Its  overall  status.  Each 
rater  will  provide  these  overall  personnel  status  ratings  for  100  mock 
ships.  Appendix  4  contains  the  complete  policy  capturing  task. 

The  1400  component  scores  were  generated  by  a  computer  program  developed 
by  Rosse  (1970)  for  the  CDC  6600.  All  scores  are  whole  numbers  ranging 
from  I  to  9.  Each  of  the  14  component  variables  has  a  distribution  close 
to  that  used  in  the  stanine  system  (across  100  ships,  mean  =  5.0,  stan¬ 
dard  deviation  ■  2.0).  Also,  the  correlation  matrix  representing  the  re¬ 
lationships  among  components  for  the  policy  capturing  task  is  approxi¬ 
mately  equivalent  to  the  correlation  matrix  generated  by  officer  raters  In 
rating  task  #2  described  earlier.  That  Is,  the  correlations  among  compo¬ 
nents  for  these  mock  ships  Is  very  similar  to  the  estimates  of  "real" 
component  intercorrelations  provided  by  officer  raters  during  the  Monterey 
workshop  meetings.  Table  5  presents  the  correlation  matrix  associated 
with  the  component  scores  In  the  policy  capturing  task.  Notice  that  the 
matrices  in  Tables  3  and  5  are  quite  similar.  Thus,  we  have  succeeded 
in  making  the  rating  task  "realistic"  In  the  sense  that  actual  relation¬ 
ships  among  components  for  our  mock  ships  map  very  closely  the  perceived 
relationships  reported  by  the  experienced  officers  who  participated  in 
the  Monterey  workshop  sessions. 

b.  Policy  Capturing  Data  Analysis 

After  the  50  officers  have  performed  the  policy  capturing  ratings,  we  will 
analyze  the  resulting  data  with  the  following  two  major  objectives  in 
mind: 

(a)  to  derive  inferences  about  the  importance  ofindividual  components 
in  terms  of  the  way  each  influences  the  overall  personnel  status 
ratings  of  mock  ships. 

(b)  to  assess  the  homogeneity  of  the  policies  generated  by  Naval 
officers. 

In  general,  the  data  analysis  will  involve  computing  multiple  regression 
equations  for  each  officer  in  the  rater  group.  Each  equation  will  repre¬ 
sent  that  rater's  policy  related  to  assessing  the  personnel  status  of 
ships.  Then,  we  will  use  JAN  and  perhaps  the  pm  statistic  to  assess  the 
similarity  in  raters'  policies.  Other  analyses  will  be  performed  as  neces¬ 
sary  to  develop  Inferences  about  the  nature  of  the  officer  groups'  compos¬ 
ite  policy,  and  about  the  Importance  of  each  component  in  Naval  officers' 
minds  for  determining  the  state  of  the  personnel  subsystem  aboard  ships. 


TABLE  5 

INTERCORRELATIONS  AMONG  COMPONENTS  FOR  THE  100  MOCK  SHIPS 
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After  the  policy  capturing  analyses  have  been  completed,  PDI's  staff 
will  select  from  the  list  of  14  those  components  which  look  most  prom¬ 
ising  for  further  study.  One  of  the  main  criteria  for  selecting  com¬ 
ponents  at  this  stage  will  be  the  magnitude  of  the  Importance  weights 
attached  to  component  measures  by  the  policy  capturing  procedure.  Thus, 
the  policy  capturing  analyses  will  result  in  a  final  list  of  components 
seen  as  possessing  maximum  potential  for  providing  meaningful  information 
within  the  NPSI  framework.  Those  components  surviving  the  policy  captur¬ 
ing  screening  should  definitely  be  of  significant  importance  in  experi¬ 
enced  officers'  minds  for  indicating  the  standing  of  a  ship's  personnel 
subsystem.  In  fact,  by  accepting  for  the  NPSI  those  components  which 
carry  weight  with  experienced  Naval  officers  in  the  way  they  assess  ships' 
personnel  status,  we  insure  that  the  areas  covered  by  our  NPSI  are  of 
practical  and  operational  significance.  The  next  step  is  to  specify  more 
exactly  how  each  component  is  to  be  measured.  Below,  we  discuss  that  step 
and  others  focused  on  the  future  development  and  validation  of  an  NPSI. 


c.  Subsequent  Requirements  for  Developing  and  Validating  an  NPSI 


For  purposes  of  explaining  our  post-policy  capturing  approach  to  NPSI 
development,  i t  wi 1 1  be  helpful  to  distinguish  among  three  different  steps 
in  the  total  NPSI  development  process. 


1)  selecting  components; 

2)  norming  component  measures; 

3)  validating  the  index. 


First,  policy  capturing  results  should  suggest  the  most  likely  components 
for  an  NPSI.  Components  selected  for  this  NPSI  will  be  those  which  are 
most  consistently  influential  in  affecting  personnel  status  evaluations 
in  the  policy  capturing  rating  task;  i.e.,  those  components  which  are 
Inferred  to  be  most  important  for  NPSI  measurement  based  on  policy  cap¬ 
turing  results.  Policy  capturing  procedures  and  the  rationale  for  using 
this  technique  have  been  outlined  previously. 

The  next  step^4  in  NPSI  development  will  involve  norming  or  scaling  each 
component  measure  included  in  the  NPSI.  To  understand  how  norming  relates 
to  policy  capturing  in  our  NPSI  development  strategy,  we  must  first  out¬ 
line  two  assumptions  made  in  the  policy  capturing  phase.  The  first 
assumption  is  that  ships  can  be  graded  fairly  (from  1  ■  extremely  poor  to 
7  ■  extremely  good)  on  each  component  measure  appearing  in  the  rating 
format. 

St  may  be  necessary  first  to  perform  varying  degrees  of  Instrument  de¬ 
velopment  work  on  components  selected  for  the  final  NPSI.  For  example,  if  the 
component  SAT-PA  is  chosen,  we  will  need  to  develop  a  questionnaire  to  tap 
that  component.  On  the  other  hand,  means  for  measuring  components  such  as 
Enlist-FSt  are  already  available. 
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That  is,  we  assume  that  some  reasonable  method  can  be  dev.sed  for  assign¬ 
ing  scale  scores  to  raw  component  scores  to  represent  fairly  the  standing 
of  ships  along  individual  components.  A  related  assumption  Is  that  offi¬ 
cers  can  eventually  agree  in  their  assignments  of  scale  scores  (I  -  7) 
across  the  complete  range  of  raw  scores  possible  on  each  component.  For 
example,  we  assume  all  officers  would  agree  that  a  certain  unauthorized 
absence  rate  (UAR)  for  a  ship  should  be  described  as  a  "A"  (average)  In 
terms  of  the  level  of  UAR  represented  by  that  raw  score. 

We  feel  that  the  first  of  these  two  assumptions  is  justified.  We  will 
outline  a  general  norming  strategy  which  should  be  effective  in  scaling 
raw  component  scores  such  that  they  reflect  properly  the  state  of  the  per¬ 
sonnel  subsystem  along  each  component.  Making  the  second  assumption  is 
necessary  to  keep  separate  the  policy  capturing  and  the  norming  activities. 
We  see  this  separation  as  extremely  desirable,  it  seems  reasonable  to 
assume  that  experienced  officers  can  agree  about  scale  score  assignments 
to  raw  scores  once  they  have  information  about  the  specific  way  in  which 
component  measures  are  to  be  taken  and  knowledge  of  the  distribution  of 
raw  scores  for  each  component  across  a  representative  sample  of  actual 
Navy  ships.  However,  officer  raters  will  not  have  this  kind  of  informa¬ 
tion  during  the  policy  capturing  task.  Therefore,  to  introduce  raw  scores 
as  component  score  cues  in  the  policy  capturing  task  forces  raters  to 
make  scaling  decisions  for  which  they  do  not  possess  good  information.  It 
appears  more  reasonable  to  provide  component  scale  scores  in  the  policy 
capturing  task  and,  thus,  to  assume  that  officers  can  agree  on  raw  scale 
score  conversions  once  they  possess  more  knowledge  of  the  actual  component 
measures  and  of  the  raw  score  distributions  across  Navy  ships.  Further¬ 
more,  including  scale  scores  rather  than  raw  scores  in  the  policy  captur¬ 
ing  rating  task  insures  that  the  policy  capturing  procedure  can  accomplish 
what  it  was  designed  to  accomplish.  Different  raters  are  able  to  make 
decisions  about  the  personnel  status  of  mock  ships  based  on  identical 
knowledge  of  component  scores  for  those  ships.  Thus,  inferences  about  the 
importance  of  components  for  NPSI  measurement  can  be  made  knowing  that 
raters  had  available  the  same  cues.  And,  the  scaling  or  norming  of  raw 
component  scores,  an  Important  activity  in  its  own  right,  does  not  con¬ 
found  the  policy  capturing  results,  instead,  norming  is  dealt  with  sepa¬ 
rately. 

The  norming  process  requires  that  experienced  Nas  i  officers  inspect  the 
distribution  of  raw  scores  for  each  component  measure  across  a  represen¬ 
tative  sample  of  real  ships.  These  officers  may  then  convert  the  compo¬ 
nent  raw  scores  to  perceived  scale  scores.  For  example,  in  dealing  with 
the  component  Unauthorized  Absence  Rate  (UAR),  officers  would  assign  each 
of  the  seven  scale  scores  to  a  range  of  raw  scores  on  the  basis  of  informa¬ 
tion  provided  by  the  raw  score  distribution  and  their  beliefs  about  the 
relative  seriousness  of  different  levels  of  UAR.  We  anticipate  that 
officers  will  not  be  able  to  make  scale  assignments  strictly  on  the  basis 
of  the  distribution  of  raw  scores.  That  is,  the  scale  score  "7"  will  not 
be  assigned  necessarily  to  the  range  of  raw  scores  representing  the  top 
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two  percent  of  the  distribution  of  scales,  the  scale  score  "6"  to  the 
range  of  raw  scores  representing  the  next  highest  eight  percent  (normal 
distribution  percentages)  in  the  raw  score  distribution,  etc.  Instead, 
officers  will  need  to  use  their  shipboard  experience  in  addition  to  in¬ 
formation  provided  by  the  raw  score  distributions  to  establish  reasonable 
norms.  Completion  of  this  norming  process  will  result  in  a  standardized 
system  for  obtaining  NPSI  scores  for  ships.  Next,  norm  tables  can  be  de¬ 
veloped  to  transform  raw  component  scores  Into  scale  scores.  Then,  for 
any  given  ship,  the  scale  scores  can  be  added  together  to  provide  an  over¬ 
all  NPSI  score  for  that  ship. 

The  third  and  final  stage  in  NPSI  development  involves  the  validation  of 
the  NPSI  composite. 5  We  must  demonstrate  that  scores  on  the  index  are 
reasonably  reliable  and  that  they  are  related  (in  the  proper  direction) 
to  events,  activities,  policy  changes,  etc.,  which  should  affect  a  ship's 
personnel  status,  it  is  also  important  to  show  that  the  NPSI  is  not 
related  significantly  to  random  events  which  should  have  nothing  to  do 
with  a  ship's  standing  on  the  index.  Essentially,  we  are  describing  the 
construct  validation  process.  Although  the  criteria  are  very  stringent 
for  comprehensively  validating  an  instrument  within  the  construct  valida¬ 
tion  framework,  this  strategy  seems  most  appropriate  for  assessing  the 
quality  of  information  provided  by  the  NPSI.  Thus,  we  recommend  perform¬ 
ing  a  "for  research  only"  empirical  validation  of  the  NPSI  following  a 
construct  validation  strategy. 


5our  factor  analysis  results  (p.  18)  suggest  that  it  might  be  reasonable 
to  represent  the  personnel  status  domain  with  two  or  more  subscales,  each  cover¬ 
ing  a  different  portion  of  this  domain.  If  multiple  NPSI  subscales  are  estab¬ 
lished,  the  same  validation  principles  discussed  in  this  section  will  apply  to 
each  subscale. 
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AGENDA  FOR  PERSONNEL  STATUS  INDEX  DEVELOPMENT  SESSIONS 


Thursday,  17  January,  1 974 


Time 

1500 

1530-1630 

1630-1630 

1830-1900 

1900-1930 

1930 


Topic 

Total  Group:  Introduction  of  panel 
participants  and . Personnel  Decisions' 
staff  members 

Total  Group:'  Genera]  Introduction  to 
the  personnel  status  Index  (PSI)  project 
including  a  description  of  progress  to 
date 

Two  Separate  Groups:  "Brainstorming" 
sessions  to  explore  present  and  potential 
measures  likely  tc  be  useful  for  the  PSI 
Group  A: 

Group  B: 

Total  Group:  Discussion  of  separate 
group  accomplishments  and  debriefing  of 
panel  participants 

Cocktai Is 

Dinner 


0800-0830 


Friday,  18  January,  197*» 

Total  Group:  Review  of  progress  and 
introduction  to  the  rating  task 


0830~1 100  Total  Group:  Perform  the  rating  task 

(approximately)  (working  individually) 


Discussant 

Dunnette  & 
Borman 


Dunnette  S 
Borman 


Dunnette 

Borman 

Dunnette  & 
Borman 


Dunnette  & 
Borman 

borman 
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WORKSHOP  PARTICIPANTS 
Roster 


A 1 1 en ,  Galen  B. 

Commander 

Bedow,  Robert  J. 

Lieutenant 

Commander 

Bel  1 ,  Mer 1 1n  G. 

Lieutenant 

Commander 

Bened 1 ktsson ,  Philip  W. 

Commander 

Browne,  Peter  A. 

Lieutenant 

Commander 

Brunei  1 1 ,  John  F. 

Lieutenant 

Daeschner ,  William 

E. 

Lieutenant 

Commander 

Emerson,  George  A. 

Lieutenant 

Glover ,  William  F. 

H.  Ill 

L 1  eutenant 

Commander 

Hamilton,  Gerald  K. 

L I eutenant 

Commander 

Larson,  Jerold  J. 

Commander 

Hay,  Douglas  E. 

Lieutenant 

Commander 

Mozingo,  Thomas  P. 

Lleutena.it 

Nemmers,  Robert  S. 

L 1  eutenant 

Commander 

Olson,  Stephen  R. 

Lieutenant 

Paddock,  John  S. 

L I eutenant 

Quinlan,  John  H. 

L 1  eutenant 

Rantschler,  Robert 

D. 

Lieutenant 

Commander 

Schraeder,  John  Y. 

L 1 eutenant 

Commander 

Stephan,  Robert  A. 

Commander 

Swan,  James  W. 

L 1 eutenant 

Commander 

Tate,  Thomas 

Commander 

Wylie,  Wa 1  ter  J . 

Lieutenant 

Commander 
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A  BRIEF  SYNOPSIS  OF  THE  NAVAL  PERSONNEL  STATUS  PROJECT 


In  late  1972,  we  proposed  a  planning  study  to  determine  the  relative 
feasibility  of  various  approaches  for  developing  a  measurement  system  for 
use  in  determining,  at  any  given  point  in  time,  the  status  of  the  Navy's 
personnel  subsystem.  For  want  of  any  other  existing  label  or  designa¬ 
tion,  we  referred  to  the  proposed  measurement  system  as  a  Naval  Personnel 
Status  Index  (NPSI).  It  was  beiieved  that  availability  of  an  NPSI  would 
aid  in  coping  with  rapid  changes  confronting  the  Navy,  changes  which  can 
be  expected  to  accelerate  with  the  advent  of  ail  volunteer  forces.  If 
developed  in  such  a  way  that  variations  in  its  magnitude  couid  yield 
information  quickly  about  the  effects  of  changes  in  Navy  personnel 
policies  and  practices,  an  NPSI  couid  provide  provide  a  continuing  audit 
or  tracking  of  the  overall  personnel  condition  of  the  Navy.  Obviously, 
such  an  index  was  not  currently  "iying  on  the  shelf"  ready  for  immediate 
application.  Its  development  would  need  to  be  undertaken  with  sophisti¬ 
cation,  capitalizing  on  the  best  available  wisdom  from  Naval  and  behavioral 
science  resources. 

A  first  step  designed  to  capitalize  on  such  wisdom  was  an  informal  meeting 
of  a  small  group  of  experts  representing  the  behavioral  sciences  and  the 
Navy.  We  anticipated  that  the  participants  in  this  meeting  would  define 
the  problem  more  explicitly,  comment  on  the  relevance  of  their' existing 
research  to  its  solution,  and  point  us  toward  other  investigators  pursuing 
research  along  similar  dimensions. 

The  meeting  occurred  on  16  and  17  November  1972  at  the  Washingtonian  Inn, 
Gaithersburg,  Maryland.  The  participants  included  the  following  persons: 


Bond,  Nicholas  A. ,  Jr. 

Department  of  Psychology 

Sacramento  State  College 

Bryan,  Glenn  L. 

Director,  Psychological  Sciences  Division 
Office  of  Naval  Research 

Campbel 1 ,  John  P. 

Department  of  Psychology 

University  of  Minnesota 

Denicoff,  Marvin 

Director,  Information  Systems  Program 
Office  of  Naval  Research 

Dunnette,  Marvin  D. 

President 

Personnel  Decisions,  Inc. 

Flamholtz,  Eric 

Graduate  School  of  Business 

Columbia  University 

Hirsch,  Warren 

Department  of  Mathematical  Statistics 

New  York  University 

Mol  berg,  Andrea 

Research  Assistant 

Personnel  Decisions,  Inc. 

2. 


Motowldlo,  Stephan  J. 
Sechrest,  Lee 
Sinaiko,  Wallace 


Research  Assistant 
Personnel  Decisions,  Inc. 

Department  of  Psychology 
Northwestern  University 

Research  Study  Director 
Smithsonian  Institution 


Smith,  Frank  Director,  Employee  Atti tude  Surveys 

Sears  Roebuck  &  Company 

Thompson,  Gerald  Graduate  School  of  Industrial  Administration 

Carnegie  Mellon  University 


Prior  to  the  meeting,  Bryan  and  Dunnette  had  each  speculated  about  the 
so-called  "ideal"  properties  that  a  Naval  Personnel  Status  Index  might 
possess.  Dunnette,  for  example,  had  written  that  such  an  index: 

‘might  be  a  single  index,  or  it  may  be  several  (such  as  economic 
indicators),  or  it  may  be  a  composite.  It  should  be  expressed  as 
a  common  number  (such  as  0  to  100,  or,  perhaps  with  100  set  as 
a  baseline)  and  should  reflect  ...  a  kind  of  combination  of 
operational  readiness,  overall  quality,  efficiency,  cost,  and 
current  level  of  'success'  of  the  personnel  subsystem  of  any 
given  Naval  organizational  unit.  [Some  measures  currently  used 
but  poorly  investigated  are  reenlistment  rates,  total  number  of 
billet  (job)  vacancies,  and  various  attitude  and  opinion  ind.ca- 
tors  from  periodic  Navy-wide  questionnaire  surveys.  If  just  a 
single  unit,  such  as  a  destroyer,  is  considered,  its  total  range 
of  possible  missions  can  be  listed  and  ordered  according  to  criti 
cality  and  probability  of  occurrence.  Naturally,  carrying  out 
most  such  missions  doesn't  require  all  personnel  aboard  the 
craft,  but  some  few--usually  of  high  cri tical i ty--do.  How  'lean' 
can  the  personnel  subsystem  of  a  destroyer  be  and  still  'get  by' 
when  the  occasional  crisis  occurs?]" 


"Such  an  index  should  be  both  robust  and  sensitive  ....  Its 
magnitude  should  not  be  subject  to  incidental  or  random  events 
.  .  .  nor  dependent  upon  complicated  parametric  assumptions.  On 
the  other  hand,  it  should  be  capable  of  reflecting  important  .  . 
changes  in  Naval  policies  and  practices.  Information  making  up 
an  NPSI  ought  to  be  easily  and  readily  available  so  that  a  con¬ 
tinuing  audit  (probably  on  a  sampling  basis)  can  be  maintained 
over  time.  However,  it  should  not  be  "fakeable"  or,  if  fakeable, 
the  methodology  must  somehow  be  such  as  to  avoid  inducing  faking. 
.  .  .  .  Certainly,  low  values  on  the  NPSI  should  not  be  viewed  as 
leading  to  punitive  actions,  though  the  head  of  an  organizational 
unit  should  view  low  values  'with  concern'  and,  in  time,  have 
capability  for  upgrading  them." 


3. 


The  results  of  the  Gaithersburg  meeting  and  a  brief  review  of  relevant 
literature  are  included  in  the  Interim  Technical  Report  [Possible  Approaches 
for  Development  of  a  Naval  Personnel  Status  Index  (NPSI)]  which  each  of 
you  has  received.  Of  the  many  approaches  reviewed  in  that  report,  each 
demands  human  Judgment.  No  approach  or  Index,  no  matter  how  objective 
appearing  nor  how  complete  its  degree  of  quantification,  is  free  from 
human  and  potentially  subjective  judgment  during  some  stage  of  its 
development . 

One  promising  approach  is  characterized  by  methods  which  have  yielded 
summary  estimates  of  the  so-called  quality  of  life  in  different  states. 

We  wish,  during  these  brief  sessions  with  you,  to  investigate  the  feasi¬ 
bility  of  using  such  an  approach  in  our  Naval  Personnel  Status  Index 
Project.  We  are  asking  the  participants  in  these  meetings  to  share  with 
us  the  wisdom  of  their  Naval  experiences  and  opinions  about  what  factors 
seem  to  be  particularly  Important,  relevant,  and  meaningful  as  indicators 
of  good  or  poor  personnel  status  in  the  Navy.  We  wish  to  use  the  infor¬ 
mation  you  give  us  to  help  in  deciding  which  measures  may  be  feasible  and 
available  from  Naval  units  on  a  systematic  and  continuing  basis.  Those 
that  are  deemed  to  be  capable  of  such  measurement  will  then  be  introduced 
on  an  experimental  or  pilot  basis  in  a  number  of  Naval  units,  and  values 
of  the  index  will  be  compared  with  other  information  ava i labl e. f rom  those 
same  units. 


W/ 
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A  CHECK  LIST  OF  SOME  POSSIBLE  INDICATORS  OF  PERSONNEL  STATUS 


accidents  and  injuries 
amount  of  agreement  among 
members  of  a  unit 
amount  of  downtime 
amount  of  effort 
amount  of  work  output 


conflict  among  members  of  a  unit 
costs  of  keeping  unit  operational 
decisiveness  of  a  unit 
difficulty  of  job  assignment 
discipl inary  actions 


disturbances  in  the  unit 
efficiency  of  the  unit 
external  stress  for  a  unit 
feedback  in  a  unit 
following  formal  chain  of  command 


formal  evaluation  of  the  unit 
griping  and  grievances 
importance  of  job  assignment 
interchangeability  of  tasks 
among  unit  members 
job  performance  ratings 


knowing  what's  required  to 
get  assignments  done 
leadership  in  a  unit 
level  of  training 
liking  for  other  people  in 
the  unit 

measures  of  life  satisfaction 


participation  rates  in  various 
programs 

pride  of  people  for  their  unit 
problem  solving  and  trouble¬ 
shooting  skills 
quality  of  work  output 
quickness  of  response  to 
assignments 


rate  of  turnover  in  the  unit 
reenlistment  rate 
reputation  of  a  unit 
scores  on  qualifying  tests 
sickness 


stabil ity  of  the  unit 
unauthorized  absences  from  unit 
unit  performance  ratings 
use  of  drugs 

willingness  to  do  whatever  is 
necessary 


measures  of  satisfaction 
measures  of  unit  morale 
military  bearing  and  appearance 
number  of  fully  qualified 
personnel  in  unit 
openness  in  a  unit 
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CRITERIA  FOR 

ASSESSING  THE  "GOODNESS"  OF  COMPONENT  MEASURES 


Importance:  Importance  attached  to  the  measure  for  determining  a 
s hi p * s  personnel  status. 


Re  1 i ab I i i ty  :  Degree  to  which  the  component  measure  is  free  from 
distortion  or  error  (either  constant  error  or  variable  error). 


Aval  lability:  Degree  to  which  the  component  measure  is  available 
or  accessible  to  researchers. 


General i zabi 1 1 ty :  Ability  of  the  component  measure  to  be  applicable 
to  a  variety  of  ship  types. 


“Fudgeab? 1 i ty11:  Degree  to  which  a  measure  of  this  component  can 
be  fudged  or  '‘gun-decked"  by  a  ship’s  personnel. 


I 
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POSSIBLE  INDICATORS  OF  PERSONNEL  STATUS 


1. 

(REENLIST) 

2. 

(MAST) 

3. 

(R/NR) 

k. 

(Q/NQ) 

5. 

(UAR) 

6. 

(REENLIST-FT) 

7. 

(TURNOVER) 

8. 

(COURT) 

9. 

(MANNING  NEC) 

10. 

(PASS  RATE) 

11 . 

(REC/EUGIBLE) 

12. 

(COURSES) 

13. 

(GROUP) 

1i*. 

(SICK  RATE) 

(SACK  TIME) 

15. 

16. 

(MORALE) 

17. 

(PERS  CAPABILITY) 

18. 

(LEADERSHIP) 

19. 

(FOOD) 

20. 

(DRUGS) 

21. 

(ADMAT) 

22. 

(DISCHARGE) 

23. 

(GRIEVANCE) 

2A. 

(MAINTENANCE) 

25. 

(TRAINED) 

26. 

(TRANSFER) 

27. 

(SAT-PA) 

28. 

(SAT-SM) 

29. 

(ENL 1  ST-FIT) 

Reenlistment  rate 

Non-judicial  punishment  rate 

Ratio  of  rated  to  non-rated  personnel 

Ratio  of  qualified  to  non-qual if  led  personnel 

Unauthorized  absence  rate 

First  tour  reenlistment  rate 

Long-term  stability  of  personnel 

Court  martial  rate 

Manning  level  ratlo--descrlblng  the  proportion  of  billets 
manned  by  qualified  persons  according  to  the  NEC  daia 
system 

Percent  of  persons  taking  rating  exams  who  pass 
Percent  of  persons  who  are  eligible  by  longevity  who  are 
recommended  for  rating 
Rate  of  correspondence  course  participation 
Rate  of  participation  in  group  activities 
Number  of  times  per  man  per  ship  per  month  sick  call  reports 
Percent  time  spent  sleeping  while  off  duty 
Rating  of  crew  morale  by  top  officers 
Overall  rating  of  personnel  capability  by  top  officers 
Perceived  leadership  effectiveness  by  crew  members 
Food  and  personal  services  evaluation 
Reported  drug  usage 
Overall  Admat  inspection  grade. 

Discharges  other  than  honorable  (Including  reference  to 
special  BuPers  Code) 

Grievances  directed  to  others  outside  of  command  (e.g., 
Congress) 

Percentage  of  maintenance  actions  deferred  due  to  insuffici¬ 
ent  manning  or  expertise  (from  3M  data  system) 

Percent  of  required  training  completed  by  officers 
Requests  for  transfer, 'man/un it  time 

Satisfaction  with  present  assignment  (as  measured  by  con¬ 
fidential  questionnaire  to  sample  of  officers  and  en- 
I 1 sted) 

Satisfaction  with  shipmates  (as  measured  by  confidential 
questionnaire  to  sample  of  officers  and  enlisted) 

GCT  +  ARI  +  Educational  Level  +  Mean  Semi-Annual  Evaluation 
(all  scores  standardized)  for  all  or  for  a  sample  of 
enlisted  persons  averaged  across  the  unit 


»  A 

A  P_  P_E  N  0  j_  X_  l 
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The  rating  task  in  this  packet  requires  you  to  make  Judgments 
about  the  "goodness"  of  the  component  measures  we  have  iden¬ 
tified  during  this  conference.  We  ask  you  to  make  your 
ratings  along  a  number  of  criteria  relevant  to  judging  the 
potential  goodness  of  the  components. 


Importance 


Importance  you  attach  to  this  measure  for  determining  a  ship's 
personnel  status. 

5  ■  very  important;  crucial  indicator 
k  ■  quite  important 

3  ■  Important  but  other  component  measures  may  be  more 
important 

2  *  of  some  importance,  but  other  component  measures  are 
definitely  more  important 

1  *  of  little  importance 


Reliability 


Degree  to  which  the  component  measure  Is  free  from  distortion 
or  error  (either  constant  error  or  variable  error). 


4  ■  extremely  error  free;  very  reliable  measure 

3  ■  might  be  subject  to  some  error  but  should  be  a 
relatively  pure  measure 

2  ■  definitely  contains  some  error,  but  the  distortion  is 
not  great  enough  to  cause  a  serious  lack  of  reliability 

1  ■  contains  so  much  error  that  scores  on  the  component 
measure  are  seriously  distorted 


Generali zabi 1 Ity 


Ability  of  the  component  measure  to  be  applicable  to  a  variety 
of  ship  types. 


2  -  completely  gener.il  izabl  e;  pertains  to  all  ship  types 

1  -  component  measure  would  not  be  general izable  to  all 
ship  types  (state  the  ship  types  for  which  the  com¬ 
ponent  measure  would  apply) 


Aval labl 1 1 ty 


Degree  to  which  the  component  measure  Is  available  or  accessible 
to  researchers. 


k  -  measure  of  this  component  is  available  now- -routinely 
gathered  information 

3  ■  easy  to  get  or  develop  such  a  component  measure 

2  "with  considerable  effort,  one  could  develop  a  measure 
for  this  component 

I  -  very  difficult  or  impossible  to  develop  a  measure  for 
this  component 


"Fudgeablllty" 


Degree  to  which  a  measure  of  this  component  could  be  fudged 
or  "gun-decked"  by  a  ship's  personnel. 


k  -  Impossible  to  fudge 
3  ■  difficult  to  fudge 

2  -  could  be  fudged  with  some  prior  knowledge  that  the 
measure  was  to  be  taken 


1 


very  easily  fudged 


"Favorable"  Set 


With  the  rating  procedure  below  we  are  attempting  to  determine  trio  degree 
to  which  components  might  be  correlated  or  might  tend  to  go  together  In 
the  real  world;  l.e.,  for  a  ship.  In  performing  these  ratings,  we  ask  you 
to  consider  a  ship  which  possesses  a  very  high  or  favorable  "score"  on  a 
particular  component.  (We  will  specify  the  components  you  have  been 
assigned  in  a  moment.)  For  example,  If  the  component  Is  "number  of  AWOLs 
per  person  on  the  ship  per  year,"  Imagine  a  ship  which  stands  extremely 
favorably  along  this  component.  With  this  hypothetical  ship  in  mind,  we 
would  like  you  to  rate  each  other  component  In  terms  of  the  likelihood 
that  the  ship  would  show  up  favorably  on  those  components.  Please  use  the 
following  code: 

9^  Certain  or  almost  certain  the  ship  would  stand  favorably  on  this 

8  c  component;  based  on  my  experience,  standing  on  these  two  compo- 

7 J  nents  Is  definitely  correlated  positively. 

A  Standing  along  this  component  should  have  little  if  anything  to 
cl  do  with  a  ship's  very  high  standing  on  the  other  component; 

Jjl  based  on  my  experience,  the  two  components  are  weakly  correlated, 

*  if  at  all. 

3}  Certain  or  almost  certain  the  ship  would  stand  unfavorably  on  this 

2f  component;  based  on  my  experience,  standing  on  these  two  compo- 

1)  nents  is  definitely  correlated  negatively. 

blank:  Don't  know  or  not  familiar  with  one  or  both  components  being  con¬ 
sidered. 


********** 


Use  the  three-point  range  within  each  category  to  express  finer  distinctions. 
For  example,  a  7  rating  would  mean  that  a  ship's  standing  along  that  compo¬ 
nent  might  tend  to  be  favorable,  but  that  It  is  by  no  means  a  certainty. 


"Unfavorable"  Set 


With  the  rating  procedure  below  we  are  attempting  to  determine  the  degree 
to  which  components  might  be  correlated  or  might  tend  to  go  together  in  the 
real  world;  i.e.,  for  a  ship.  In  performing  these  ratings,  we  ask  you  to 
consider  a  ship  which  possesses  a  very  low  or  unfavorable  "scorg"  on  a 
particular  component.  (We  will  specify  the  components  you  have  been  assigned 
in  a  moment.)  For  example,  if  the  component  is  "number  of  AWOLs  per  person 
on  the  ship  per  year,"  imagine  a  ship  which  stands  extremely  unfavorably 
along  this  component.  With  this  hypothetical  ship  in  mind,  we  would  like 
you  to  rate  each  other  component  In  terms  of  the  likelihood  that  the  ship 
would  show  up  favorably  on  those  components .  Please  use  the  following  code: 

91  Certain  or  almost  certain  the  ship  would  stand  favorably  on  this 
8r  component;  based  on  my  experience,  standing  on  these  two  components 
7 )  is  definitely  correlated  negatively. 

61  Standing  along  this  component  should  have  little  if  anything  to  do 
5f  with  a  ship's  very  low  standing  on  the  other  component;  based  on 
AJ  my  experience,  the  two  components  are  weakly  correlated,  if  at  all. 

3J  Certain  or  almost  certain  the  ship  would  stand  unfavorably  on  this 
2r  component;  based  on  my  experience,  standing  on  these  two  components 
lj  is  definitely  correlated  positively. 

blank:  Don't  know  or  not  familiar  with  one  or  both  components  being  con¬ 
sidered. 


********** 

Use  the  three-point  range  within  each  category  to  express  finer  distinctions. 
For  example,  a  7  rating  would  mean  that  a  ship's  standing  along  that  compo¬ 
nent  might  tend  to  be  favorable,  but  that  it  is  by  no  means  a  certainty. 


Total  Component  List 


Components  being  "rated 


I  s~a 

I 

I 

i 

I 

I 
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INSTRUCTIONS  FOR  THE  RATING  TASK 

Today  we  are  asking  you  essentially  to  evaluate  the  state  of  the  personnel 
subsystem  for  each  of  100  ships.  Each  "ship"  will  be  a  hypothetical  one, 
described  by  scale  scores  on  14  components  thought  to  be  related  to  the 
state  of  the  personnel  subsystem  or  to  the  personnel  status  of  that  ship. 


As  you  will  see  in  a  moment,  your  task  Is  to  study  the  14  component  scores 
assigned  to  each  ship  and  then  to  rate  each  of  these  ships'  personnel  status 
based  on  your  assessment  of  the  component  scores.  Here  are  the  14  components 

1.  (REENLIST)  Reenlistment  rate 

2.  (HAST)  Nonjudiclal  punishment  rate 

3.  (UAR)  Unauthorized  absence  rate 

4.  (REENLIST-FT)  First  tour  reenlistment  rate 

5.  (TURNOVER)  Long-term  stability  of  personnel 

6.  (MANNING-NEC)  Manning  level  ratio  describing  the  proportion  of 
billets  manned  by  qualified  persons  according  to  the  NEC  data 
system 

7.  (PASS  RATE)  Percentage  of  persons  taking  rating  exams  who  pass 

8.  (COURSES)  Rate  of  correspondence  course  participation 

9.  (SACK  TIME)  Percent  of  time  spent  sleeping  while  off  duty 

10.  (LEADERSHIP)  Perceived  leadership  effectiveness  by  crew  members 

11.  (DISCHARGE)  Discharges  other  than  honorable  (including  reference 
to  special  BuPers  code) 

12.  (MAINTENANCE)  Percentages  of  maintenance  actions  deferred  due  to 
insufficient  manning  or  expertise  (from  3M  data  system) 

13.  (SAT-PA)  Satisfaction  with  present  assignment  (as  measured  by 
confidential  questionnaire  to  sample  of  officers  and  enlisted) 

14.  (ENLIST-FIT)  GCT  +  ARI  +  Education  Level  +  Mean  Semi-Annual 
Evaluation  (all  scores  standardized)  for  all  or  for  a  sample  of 
enlisted  persons  averaged  across  the  ship. 


Now  that  you  have  received  a  very  basic  outline  of  what  we  will  be  doing  to¬ 
day,  let's  digress  a  bit  and  supply  some  background  concerning  this  project. 
The  Office  of  Naval  Research  (ONR)  has  asked  Personnel  Decisions,  Inc.  (our 
firm)  to  move  toward  developing  a  Naval  Personnel  Status  Index.  The 


2. 


personnel  status  index  (NPSI)  is  to  be  an  indicator  of  the  state  of  the 
human  organization  or  of  the  personnel  status  of  organizational  units  such 
as  ships.  As  part  of  our  NPSi  development  effort,  we  visited  the  Navy 
Postgraduate  School  in  January  to  ask  experienced  officers  to  help  us  gen¬ 
erate  a  list  of  components  with  good  potential  for  contributing  to  NPSI 
measurement. 

The  result  of  the  January  workshops  here  was  a  list  of  29  component  measures 
thought  to  have  potential  for  indicating  the  personnel  status  of  ships.  On 
the  second  day  of  the  workshops,  officers  also  provided  evaluations  of  the 
"goodness"  of  each  component  along  five  criteria— Importance,  Reliability, 
General izabi I ity,  Accessibility,  and  Fudgeabl 1 1 ty.  Analysis  of  these  eval¬ 
uations  enabled  us  to  reduce  the  number  of  components  for  further  consider¬ 
ation  to  14.  These  14  components  seem  to  represent  a  wide  spectrum  of 
possible  measures  for  a  comprehensive  NPSI. 

Broadly  speaking,  the  purpose  of  the  rating  task  today  is  to  determine  how 
officers  use  this  information  from  components  to  assess  the  personnel 
status  of  ships.  A  more  elaborate  explanation  of  our  purposes  In  this 
rating  task  would  possibly  affect  your  evaluations;  consequently,  we  will 
defer  those  explanations  until  later.  A  "Debrief  Package"  describing  more 
completely  the  purpose  of  this  rating  session  will  be  made  available  to 
you  after  the  session. 

Let's  now  turn  to  the  rating  task  and  describe  in  detail  what  we  need  from 
you.  First,  you  should  have  the  foliowing  materials: 

a.  a  short  questionnaire 

b.  a  page  containing  a  definition  of  "personnel  status"  and  a 
description  of  each  of  the  14  components 

c.  100  "ships"  each  described  by  14  component  scores 

d.  five  "practice  ships"  each  described  by  14  component  scores 

e.  a  rating  worksheet  to  record  your  evaluations 


After  you  have  checked  your  materials,  please  fill  out  the  questionnaire  and 
then  get  ready  to  begin  the  ratings.  In  order  to  Insure  that  everyone  Inter 
prets  the  rating  task  in  a  similar  way,  we  wilt  now  discuss  a  few  ground 
rules  and  some  things  to  remember  about  the  rating  procedures. 

a.  Component  scores  are  based  on  a  (1-9)  scaie  where  1  *  extremely 
unfavorable;  5  ■  average;  and  9  ■  extremely  favorable.  It  is  very 
important  that  you  Interpret  each  component  score  along  a  favorable- 
unfavorable  continuum.  This  will  enable  you  to  forget  about  the 
fact  that  "high"  on  some  components  is  good,  but  on  other  components 
"low"  is  good.  For  example,  a  "high"  reenlistment  rate  Is  "good," 
while  a  "how"  nonjudicial  punishment  rate  is  "good."  You  need  not 
worry  about"1 ''high"  and  "low"  In  that  sense.  Instead,  remember  that 
the  component  scores  relate  only  to  how  favorable  ships  stand  along 
these  components. 


3. 


b.  When  considering  individual  component  scores  for  a  ship,  assume 
that  each  score  was  derived  in  a  reasonably  competent  manner. 

Let's  use  two  examples  of  components  not  included  In  the  list  of 
14  in  order  to  illustrate  this  point.  For  the  component  "Rate 
of  Participation  in  Group  Activities,"  you  should  assume  that  a 
reasonably  good  definition  of  "Group  Activities"  could  be  gener¬ 
ated  and  that  the  measure  of  the  rate  of  participation  in  these 
activities  was  done  properly.  For  the  component  "Satisfaction 
with  Shipmates,"  you  should  assume  that  the  questionnaire  designed 
to  measure  this  component  would  be  reasonably  well  constructed  and 
that  the  persons  aboard  each  ship  being  assessed  would  be  properly 
sampled  and  would  do  a  reasonably  conscientious  job  of  completing 
the  quest ionnai re. 

In  other  words,  assume  that  the  scores  assigned  to  the  ships  with¬ 
in  this  rating  task  are  reasonable  estimates  of  the  "true"  scores 
for  these  components  (to  the  limits  of  the  various  measures'  capa¬ 
bilities  to  provide  uncontaminated,  "good"  data). 

c.  For  your  first  pass  through  the  ships,  we  suggest  that  you  evalu¬ 
ate  them  at  a  rate  of  approximately  one  per  minute.  Some  of  you 
may  be  able  to  assimilate  more  quickly  the  information  provided  by 
the  component  scores  and  some  may  need  longer  to  do  a  conscientious 
job  with  the  ratings.  However,  our  pretest  suggested  that  one  ship 
per  minute  is  a  reasonable  rate  to  shoot  for.  (Of  course,  the  first 
five  to  ten  ships  may  go  a  little  slower  because  you  will  still  be 
comparatively  unfamiliar  with  the  components.) 

If  you  understand  points  a,  b,  and  c  above,  you  are  ready  to  begin  evalua¬ 
ting  the  first  of  five  "practice  ships"  In  your  packet.  To  get  everyone 
off  on  the  right  foot,  we  will  now  describe  the  procedure  for  making  per¬ 
sonnel  status  evaluations.  First,  read  carefully  the  definition  of  person¬ 
nel  status.  It  is  essential  that  you  have  a  clear  idea  of  what  is  meant  by 
this  term.  Then  read  the  component  descriptions  to  get  their  general  defi¬ 
nitions  firmly  in  mind.  Now  remove  Practice  Ship  I  from  your  packet.  Study 
the  component  scores  for  that  ship.  When  you  have  formed  a  clear  impression 
of  the  status  of  the  personnel  subsystem  on  that  ship,  record  your  rating 
(from  1  ■  extremely  poor  to  7  “  extremely  good)  next  to  the  appropriate 
blank  on  the  rating  work  sheet.  Do  the  same  for  practice  ships  2-5.  If  you 
have  questions  about  the  task  at  that  time,  feel  free  to  discuss  them  with 
us.  If  you  have  no  questions,  you  are  ready  to  proceed  with  the  "for  real" 
ships. 

Here  is  the  procedure  to  follow  for  these  100  ships.  Instead  of  simply 
rating  ships,  we  ask  you  to  sort  each  ship  into  a  category  (from  1  ■ 
extremely  poor,  to  7  ■  extremely  good)  according  to  a  specific  plan.  On 
the  basis  of  your  judgments  about  the  personnel  status  of  each  ship,  you 
should  sort  ships  into  categories  to  satisfy  the  following  requirements: 


k. 


Personnel  Status  Category 


extremely  poor  -  1 
poor  -  2 
somewhat  below  average  -  3 
average  -  k 
somewhat  above  average  -  5 
good  -  6 
extremely  good  -  7 


Number  of  Ships  to  be  Sorted 
Into  the  Category _ 

2 

8 

2k 

32 

2k 

8 


To  accomplish  this  sorting  task,  we  ask  you  to  comply  with  the  following 
procedure.  For  your  first  pass  through  the  ships,  take  about  one  minute  to 
study  the  component  scores  for  each  ship.  When  you  have  decided  on  an  over¬ 
all  personnel  status  rating  for  a  ship,  record  your  evaluation  (1-7)  on  the 
work  sheet  according  to  the  number  code  on  the  ship  (e.g.,  NO  or  TV);  also, 
sort  the  "ship"  (card  containing  the  component  scores)  into  a  pile  contain¬ 
ing  other  ships  assigned  the  same  rating.  That  is,  as  you  proceed  through 
the  ships,  form  seven  stacks,  one  for  each  of  the  seven  points  on  the  rat¬ 
ing  scale.  Please  try  to  remain  roughly  within  the  guidelines  describing 
the  desired  rating  distribution  as  you  make  your  evaluations.  For  example, 
you  should  not  be  rating  every  ship  above  average. 

After  you  have  sorted  each  ship  Into  one  of  the  seven  scale  categories  and 
recorded  all  ratings  on  the  work  sheet,  count  the  number  of  ships  you  have 
placed  In  each  category.  If  your  distribution  of  ratings  Is  more  than  a 
few  off  from  the  desired  distribution,  resort  ships  to  other  categories  un¬ 
til  your  distribution  is  exactly  like  or  very  similar  to  the  distribution 
outlined  earlier.  This  operation  will  require  you  to  go  back  through  cer¬ 
tain  piles  to  find  ships  that  should  be  resorted.  After  you  have  made  all 
changes,  record  them  on  the  work  sheet.  This  step  completes  the  task. 

Below  we  provide  an  outline  or  a  "check  list"  of  the  procedures  presented 
above : 

1)  insure  that  you  have  all  required  materials 

2)  complete  the  questionnaire 

3)  read  ground  rules  on  pages  2  and  3 

k)  read  carefully  the  definition  of  personnel  status,  the  descriptions 
of  the  components,  and  the  directions  for  completing  the  ratings  and 
the  sorting 

5)  rate  the  five  practice  ships 

6)  rate  and  sort  the  100  ships 

7)  resort  ships  (If  necessary)  to  comply  with  the  desired  distribution 

8)  record  all  rating  changes  on  the  work  sheet 


The  next  page  contains  four  "ships**  extracted  from  the  rating  task.  Within 
the  task,  component  order  will  be  randomly  presented  for  each  ship;  l.e., 
the  order  of  components  will  generally  change  for  each  succeeding  ship 
presented  to  a  rater.  Also,  the  100  ships  will  be  presented  to  each  rater 
in  random  order. 


SHIP  NUMBER  001  (NO) 


COMPONENT 

SCORES 

Reenlist-ft 

i****4 

UAp 

I 

1***3 

REENLIST 

1 

l**«*4 

enlist-fit 

1 

t  ####4 

turnover 

i 

leadership 

i 

t***3 

SAT-PA 

i***3 

MAST 

1 

1**2 

manning-nec 

1 

l****4 

COURSES 

1 

1***3 

discharge 

1 

l«****5 

maintenance 

1 

l***«4 

PASS  RATE 

1 

1 **««4 

sack  time 

t 

1*1 

SHIP  NUMBER  003  (NQ) 
•COMPONENT  SCORES 

peenlist-ft 
maintenance 
reenlist 
courses 
pass  rate 
manning-nec 

SAT-PA 
MAST 

LEADERSHIP 
SACK  TIME 
TURNOVER 

discharge 

UAp 

ENLIST-FIT 


*4***5 

•••3 


«»« 

### 

••• 

••• 

*•• 

•*5 

••• 

SHIP  NUMBER  002  (NP) 


COMPONENT 

SCOPES 

maintenance 

i***«4  i 

REFNLIST 

i  i 

l***3  1 

enlist-fit 

t  1 

1**2  1 

SACK  TIME 

1  1 

manning-nec 

1  1 

1***3  1 

discharge 

S  1 

1*1  1 

MAST 

1  1 

1*1  1 

TURNOVER 

1*1  1 

PASS  RATE 

UAR 

1  1 

1*1  1 

COURSES 

1  1 

LEADERSHIP 

I  1 

!****4  1 

reenlist-ft 

I  1 

1***3  1 

sat-pa 

1  1 

SHIP  NUMBER  004  (NR) 

component  scopes 

manning-nec 

refnlist-ft 

i  i 

UAP 

9  1 

OlsCHARGE 

t  i 

courses 

t  1 

SACK  TIME 

1  1 

mast 

i  I 

ENLIST-FIT 

i  i 

LEADERSHIP 

1  1 

REENllST 

i  t 

maintenance 

i  i 

PASS  RATE 

t  1 

turnover 

i  i 

Sat-pa 

■  1 

Rating  Worksheet 


1  ■  extremely  poor  4  “ 

2  ■  poor 

3  -  somewhat  below  average 

Practice 

Ships 

1 

2 

3 

4 

5 

Personnel 
Status  Rating 
"Real"  Ships  (I  -  7) 

NO  _ 

NP  _ 

NQ  _ 

NR  _ 

NS  _ 

NT  _ 

NU  _ 

NV  _ 

NW  _ 

NX  _ 


average  5  ■  somewhat  above  average 

6  -  good 

7  ■  extremely  good 

Personnel 
Status  Rating 
0  -  7) 


Personnel 
Status  Rating 
"Real"  Ships  (1  -  7) 

ON  _ 

00  _ 

OP  _ 

0Q  _ 

OR  _ 

OS  _ 

0T  _ 

0U  _ 

0V  _ 

0W  _ 


OX 


Rating  Worksheet  (Continued) 


extremely  poor 
poor 

somewhat  below 

k  -  average 

average 

5  ■  somewhat  above  average 

6  ■  good 

7  ■  extremely  good 

"Real"  Ships 

Personnel 

Status  Rating 
(1  -  7) 

"Real"  Ships 

Personnel 
Status  Rating 
(1  -  7) 

PN 

QN 

PO 

QO 

PP 

QP 

PQ 

QQ 

PR 

QR 

PS 

QS 

PT 

QT 

PU 

QU 

PV 

QV 

PW 

QW 

PX 

QX 

"Real"  Ships 


Personnel 
Status  Rating 
(1  -  7) 


"Real"  Ships 


Personnel 
Status  Rating 
(i  -  7) 


Rating  Worksheet  (Continued) 


3 


s-f 


1  -  extremely  poor  4  ■  average  5  ■ 

2  ■  poor  6  ■ 

3  -  somewhat  below  average  7  ■ 

Personnel 
Status  Rating 

"Real"  Ships  (I  -  7)  "Real11 

TN  UN 

TO  UO 

TP  UP 

TQ  UQ 

TR  UR 

TS  US 

TT  UT 

TU  UU 

TV  UV 

TW  UW 

TX  UX 

Personnel 
Status  Rating 

"Real"  Ships  (i  -  7)  "Real" 

VN  WN 

VO  WO 


VP 

VQ 

VR 

VS 

VT 

VU 

VV 

VW 

VX 


somewhat  above  average 
good 

extremely  good 

Personnel 
Status  Rating 
Ships  (I  -  7) 


Personnel 
Status  Rating 
Ships  (1  -  7) 


QUESTIONNAIRE 


Your  present  rank  _ ___________________________ 

Your  "specialty,"  if  applicable;  e.g.,  aviator,  submariner,  destroyer 
officer,  etc.  __________ 

List  the  types  of  ships  you  have  served  aboard 


How  familiar  are  you  with  the  NtC  data  system? 

_ very  familiar;  _ familiar;  know  only  basically  what  it 

involves;  _ unfamiliar. 

How  familiar  are  you  with  the  3M  data  system? 

_ very  fami  1  lar;  _ fami  1  lar;  _ know  only  basical  ly  what  it 

involves;  _ unfamiliar. 

How  would  you  assess  your  overall  familiarity  with  administrative 
systems  and  indices  kept  in  BuPers,  aboard  ship,  and  at  various  head¬ 
quarter  levels? 

_ very  familiar;  I  have  considerable  experience  with  a  wide  variety 

of  such  systems  and  indices. 

_ quite  familiar;  1  have  at  least  average  familiarity  with  such 

systems  and  indices  compared  to  an  experienced  LCDR  or  CDR. 

_  familiar  with  most  such  systems  and  indices  with  the  exception  of 

~  the  following  areas: 


not  very  familiar;  1  have  relatively  little  experience  with  such 
systems  and  Indices. 


DEFINITION  OF  PERSONNEL  STATUS 


Personnel  Status  is  the  component  of  operational  readiness  related  to  the 
personnel  subsystem  of  a  ship.  It  Is  that  part  of  operational  readiness 
or  of  potential  unit  effectiveness  which  is  affected  directly  by  the  status 
of  the  personnel  subsystem.  Another  way  of  thinking  about  a  ship's  person¬ 
nel  status  is  that  the  ship's  overall  readiness  is  a  function  of  equipment 
capabilities  and  status,  situational  constraints,  and  the  state  of  the  huma 
organization — we  are  focusing  in  on  the  last  of  these  three  broad  factors 
related  to  operational  readiness. 


The  Components 

1.  (REENLIST)  Reenlistment  rate 

2.  (MAST)  Nonjudicial  punishment  rate 

3.  (UAR)  Unauthorized  absence  rate 

4.  (REENLIST-FT)  First  tour  reenlistment  rate 

5.  (TURNOVER)  Long-term  stability  of  personnel 

6.  (MANNING-NEC)  Manning  level  ratio  describing  the  proportion  of  billets 
manned  by  qualified  persons  according  to  the  NEC  data  system. 

7.  (PASS  RATE)  Percentage  of  persons  taking  rating  exams  who  pass 

8.  (COURSES)  Rate  of  correspondence  course  participation 

9.  (SACK  TIME)  Percent  of  time  spent  sleeping  while  off  duty 

10.  (LEADERSHIP)  Perceived  leadership  effectiveness  by  crew  members 

I).  (DISCHARGE)  Discharges  other  than  honorable  (including  reference  to 
special  BuPers  Code) 

12.  (MAINTENANCE)  Percentages  of  maintenance  actions  deferred  due  to  insuf 
ficient  manning  or  expertise  (from  3M  data  system) 

13.  (SAT-PA)  Satisfaction  with  present  assignment  (as  measured  by  confi¬ 
dential  questionnaire  to  sample  of  officers  and  enlisted) 

14.  (ENLIST-FIT)  GCT  +  ARI  +  Education  Level  +  Mean  Semi-Annual  Evaluation 
(all  scores  standardized)  for  all  or  for  a  sample  of  enlisted  persons 
averaged  across  the  ship 


Debrief  Package 

Responses  you  made  on  the  rating  task  just  completed  will  provide  us  with  a 
model  of  your  "policy"  related  to  making  Judgments  about  personnel  status. 

From  your  ratings,  we  will  generate  policies  for  Individual  raters  and  com¬ 
posite  policies  for  groups  of  raters.  The  policy  capturing  analyses  we  are 
describing  will  yield  an  importance  weight  for  each  of  the  14  components  re¬ 
flecting  the  effect  scores  on  that  component  had  on  your  overall  personnel 
>tatus  evaluations.  We  will  perform  the  same  analyses  on  the  group's  com¬ 
posite  ratings,  allowing  us  to  obtain  importance  weights  appropriate  for  the 
vhole  group  as  well. 

A  central  purpose  for  generating  these  importance  weights  is  to  provide  a 
means  for  discovering  which  component  measures  experienced  Naval  officers 
weight  heavily  when  assessing  the  personnel  status  of  ships.  In  a  sense,  we 
will  be  formalizing  and  systematizing  Naval  officer  wisdom  by  capturing  the 
policy— i.e.,  obtaining  component  importance  welghts--of  officers  participa¬ 
ting  in  this  project.  The  way  you  and  the  other  officers  use  and  combine 
this  information  about  ships'  personnel  status  will  yield  important  clues 
about  what  types  of  measures  should  be  most  legitimately  included  in  an  NPSI . 

A  rejated  objective  of  this  policy  capturing  rating  task  is  to  assess  the 
degree  of  policy  similarity  across  Naval  officers.  Data  you  provide  will 
suggest  either  that  officers  use  substantially  different  weighting  systems 
in  making  judgments  about  ships'  personnel  status  or  that  officers  regard  as 
important  much  the  same  kind  of  information  when  assessing  the  state  of  ships' 
personnel  subsystems. 

What  will  these  results  mean  to  the  future  of  the  project?  As  of  now,  we  are 
not  funded  to  proceed  further  with  our  study.  However,  if  we  receive  money 
to  go  forward,  we  would  first  choose  carefully  those  components  which  seemed 
most  important  on  the  basis  of  the  policy  capturing  data.  Then  we  would 
initiate  a  thorough  study  of  the  psychometric  properties  of  one  or  more  mea¬ 
sures  of  each  of  these  selected  components.  That  is,  we  would  focus  on  ac¬ 
tually  trying  out  various  measures  in  an  attempt  to  get  the  best  Indicator 
possible  for  each  important  component . 

Once  the  operational  measures  were  finally  chosen  (developed),  the  NPSI  com¬ 
posite  would  be  ready  for  a  trial  run.  Basically,  our  research  strategy 
would  involve  assessing  the  "construct  validity"  of  the  NPSI— i.e.,  finding 
out  whether  or  not  it  was  related  to  what  it  should  be  related  to  and  Insur¬ 
ing  that  it  was  not  related  to  other  measures  or  phenomena  having  little  to 
do  with  the  personnel  status  domain. 

Once  such  an  index  was  developed  and  validated,  there  would  exist  a  number  of 
potential  applications.  A  couple  of  examples:  An  NPSI  could  be  used  to 
assess  the  effect  of  "people  programs"  instituted  either  Navywide  or  within 
lower  level  commands.  Also,  the  index  would  be  a  valuable  diagnostic  tool 
for  a  person  with  command  responsibilities.  NPSI  scores  would  suggest  areas 
in  the  personnel  subsystem  which  most  needed  his  attention. 

In  closing,  the  Personnel  Decisions'  staff  would  like  to  thank  you  very  much 
for  participating  in  this  research.  We  pledge  to  send  one  more  newsletter 
describing  the  results  of  the  policy  capturing  rating  task  you  just  completed. 
Hope  to  see  you  again — smooth  sailing. 


